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holistic  rankings  and  ratings  of  the  three  options. 

The  experiment  had  three  basic  findings:  first,  hierarchical  weights  were 
steeper  (higher  weight  ratios)  than  non- hierarchical  weights.  Second,  groups 
identified  by  their  holistic  first  choice  showed  substantial  agreement  in  their 
assessment  of  attribute  weights.  Attribute-wise  ratings  of  energy  options  also 
agreed  rather  well  across  groups,  although  there  was  a  tendency  for  each  group 
to  favor  their  holistic  first  choice.  This  convergence  of  MAU  model  parameters 
resulted  in  a  "conmon  model"  that  was  rather  consistent  with  holistic  evalua¬ 
tions  of  the  pro -conservation  group,  and  generally  inconsistent  with  those  of 
the  pro-nuclear  group.  This  third  finding  of  differential  consistency  between 
model  composites  and  holistic  evaluations  is  interpreted  as  a  result  of  weight 
parameter  distortions  due  to  social  desirability  and  a  neglect  to  consider 
attribute  value  ranges  when  making  weight  judgments. 
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I.  Introduction 


Complex  evaluation  problems  can  be  structured  by  building  a  value 
tree  (Edwards,  1980),  objectives  hierarchy  (Keeney  and  Raiffa,  1976),  or 
analytic  hierarchy  (Saaty,  1977) .  These  are  essentially  equivalent 
semantic  structures  that  relate  general  values  and  concerns  of  the  evalu¬ 
ator  to  specific  attributes  descriptive  of  the  alternatives  under  con¬ 
sideration.  Value  trees  are  usually  displayed  in  pyramid  format  with  a 
small  number  of  general  value  labels  at  the  top  level,  several  inter¬ 
mediate  level  objectives,  and  a  larger  number  of  specific  value  relevant 
attributes  at  the  bottom  level.  The  lowest  level  attributes  (twigs) 
serve  to  explicate  and  operationalize  higher  level  values  and  concerns. 
Figure  1  shows  a  schematic  example  of  a  simple  value  tree  with  two 
objectives  and  seven  twigs. 

Insert  Figure  1  about  here 

Edwards  (1977)  proposed  the  following  procedure  for  evaluating  al¬ 
ternatives  with  respect  to  the  attributes  of  a  value  tree.  First,  the 
performance  of  each  alternative  x  is  defined  on  each  attribute  via 
single  attribute  ratings  v^(x).  Next,  a  weight  w^  is  assigned  to  each 
attribute,  reflecting  its  "relative  importance"  in  the  evaluation.  Finally, 
weights  and  single  attribute  ratings  are  combined  into  an  overall  evalu¬ 
ation  of  each  alternative  by  an  additive  model: 

n 

V(x)  =  £  w.  v.  (x)  (1) 

i=l  1  1 

While  equation  (1)  is  usually  defined  using  the  twigs  of  a  value 
tree,  alternative  procedures  exist  which  exploit  the  hierarchical 
structure  of  the  tree.  First,  equation  (1)  can  be  applied  to  higher 


i 


I  «r.  ■» »n». •  t-OMWa-W-  w 


2nd  level 
weights : 


OVERALL  VALUE 


1st  level 
weights: 


OBJECTIVE  1 


OBJECTIVE  2 


direct 
weights : 


ATTR.1  ATTR.2  ATTR.3  ATTR.4  ATTR.5  ATTR. 


Figure  1:  Schematic  Value  Tree 
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levels  of  the  tree.  In  the  value  tree  structure  of  Figure  1,  for  example, 
one  could  build  a  model  on  the  level  of  the  two  objectives,  rather  than 
on  the  level  of  the  seven  twigs.  A  second  option  is  to  assess  weights 
at  different  levels  of  the  tree,  considering  only  branches  that  belong 
to  the  same  node.  Weights  for  any  particular  level  can  then  be  obtained 
by  multiplying  these  "conditional"  weights  down  the  tree.  In  the  value 
tree  of  Figure  1,  for  example,  one  would  first  weight  attributes  1-3  with 
respect  to  objective  1,  then  weight  attributes  4-7  with  respect  to  objec¬ 
tive  2,  and  finally  assign  relative  weights  to  the  two  objectives.  Multi¬ 
plying  first  level  and  second  level  weights  would  produce  the  relevant 
twig  weights. 

To  distinguish  among  possible  combinations  of  models,  we  will  refer 
to  a  "k-th  level  model"  whenever  the  alternatives  were  rated  on  the  k-th 
level  of  attributes.  We  will  refer  to  a  "hierarchical  model"  whenever 
weights  were  obtained  by  multiplying  conditional  weights  down  the  tree. 

In  a  non-hierarchical  model,  in  contrast,  all  attributes  at  a  given  level 
are  weighted  simultaneously. 

In  general,  decision  analysts  favor  lower  level  models,  because 
single  attribute  ratings  are  simpler  for  specific  attributes.  Higher 
level  models  are  sometimes  used  for  consistency  checks,  but  it  is  assumed 
that  lower  level  models  provide  more  valid  representations  of  the  evalu¬ 
ator's  preferences.  Whether  to  use  a  hierarchical  or  non-hierarchical 
model  is  a  more  tricky  question.  Advocates  of  hierarchical  procedures 
argue  that  the  semantic  structure  of  the  value  tree  provides  a  convenient 
frame  for  canparing  values  and  objectives,  since  it  requires  importance 
judgments  only  within  relatively  small  sets  of  objectives  and  attributes  at 
comparable  levels  of  abstraction. 
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But  there  are  also  potential  problems.  Hierarchical  weighting 
assunes  that  attributes  under  each  node  are  independent  and  describe 
the  node  exhaustively.  However,  some  attribute  sets  may  not  be  inde¬ 
pendent,  and  more  abstract  attribute  labels  may  not  be  psychologically 
equivalent  to  the  set  of  more  concrete  attribute  labels  directly  below. 
Another  problem  is  that  weights  often  depend  on  the  ranges  of  alternatives 
.on  attributes.  Unfortunately,  such  ranges  become  ill-defined  and  vague 
at  higher  levels  of  the  value  tree. 

So  far  there  has  been  very  little  experimental  research  on  multi¬ 
level  and/or  hierarchical  models.  Sayeki  and  Vesper  (1973)  compared 
hierarchical  and  non-hierarchical  weighting  procedures  and  found  that 
hierarchical  weights  are  steeper  (larger  weight  ratios  between  pairs  of 
attributes  )  than  non-hierarchical  weights.  While  no  experiment  has 
systematically  explored  the  differences  between  models  built  at  different 
levels  of  the  tree,  a  number  of  experiments  have  compared  "twig"  models 
with  so-called  ,rholistic"  assessments,  in  which  the  overall  value  of 
alternatives  is  rated  directly.  Such  holistic  judgments  have  been  used 
to  derive  the  implicit  weighting  policy  (through  regression  or  other 
statistical  models)  and  these  implicit  weights  were  compared  with  directly 
assessed  weights  at  the  twig  level.  The  stable  result  of  this  research 
was  that  implicit  weights  are  steeper  than  explicit  weights  (see,  for 
example,  Slovic  and  Lichtenstein,  1971;  John  and  Edwards,  Note  1). 

One  interpretation  of  this  result  is  that  people  concentrate 
only  on  the  two  or  three  most  important  attributes  when  making  holistic 
judgments  and  neglect  the  less  important  ones.  Since  people  with 
differing  value  positions  typically  stress  the  importance  of  different 
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sets  of  attributes,  one  would  therefore  expect  more  disagreement  in 
their  holistic  judgments  than  in  their  multiattribute  model,  and 
perhaps  more  disagreement  among  higher  level  models  than  among  lower 
level  models. 

The  experiment  described  in  the  following  sections  explored  such 
differences  between  multilevel  hierarchical  and  non-hierarchical  models. 
The  experiment  was  designed  to  answer  the  following  questions: 

1)  Are  hierarchical  weights  systematically  different  from 
non-hierarchical  weights? 

2)  Are  higher  level  ratings  systematically  different  fron 
predictions  of  lower  level  models? 

3)  Do  lower  level  models  display  more  (or  less)  agreement 
among  subjects  than  higher  level  models? 

To  explore  these  questions,  we  examined  a  problem  in  which  subjects  had 
to  evaluate  three  energy  supply  options  on  13  attributes  structured  in 
the  form  of  a  value  tree.  This  problem  is  described  next. 

II.  The  Energy  Supply  Scenario  and  Value  Tree 

We  chose  a  controversial  problem  in  order  to  allow  subjects'  value 

differences  to  bear  on  the  evaluation.  The  problem  was  how  to  supply 

electricity  to  a  region  whose  population  was  growing  at  such  a  fast 

rate  that  the  regional  utility  company's  present  supply  capacity  would 

soon  fall  short  of  the  demand.  The  scenario  included  three  alternatives: 
2 

a  2400  Megawatt  (MV)  nuclear  plant,  a  2400  MW  coal  fired  plant,  and  a 


smaller  geothermal  plant  (800  MV)  with  a  combined  package  of  strict 


conservation  measures.  Each  of  the  three  proposals  was  summarized  on 
one  page,  describing  the  nature  of  the  plant  and  surrounding  environment 
in  general  terms.  It  was  stressed  that  the  geothermal  plant/conservation 
package  would  require  intrusive  conservation  measures,  such  as  direct 
restrictions  on  electricity  use  in  peak  hours,  as  well  as  more  passive 
measures  (e.g.,  tax  incentive  programs). 

Detailed  value  trees  for  comparing  such  energy  options  had  been 
developed  as  part  of  another  research  project  (see,  v.  Winterfeldt  and 
Otway,  1981) .  These  value  trees  were  elicited  from  members  of  groups 
involved  in  the  energy  supply  debate:  utility  company  executives, 
representatives  of  regulatory  agencies,  and  environmental  groups.  These 
group  specific  value  trees  were  very  detailed  and  included  up  to  100  twigs. 
For  the  purpose  of  this  experiment,  we  radically  pruned  these  trees  and 
combined  them  into  one  joint  tree,  shown  in  Figure 2.  While  we  cut  the 
tree  well  above  measurable  attribute  levels,  we  attempted  to  cover  all 

Insert  Figure  2  about  here 

concerns  mentioned  in  the  single  group  trees.  We  also  attempted  to  pro¬ 
vide  relatively  specific  descriptions  of  the  thirteen  twigs.  These 
descriptions  are  presented  in  Table  1. 

-Insert  Table  1  about  here 


III.  Method 

Sub j ects .  Subjects  were  37  undergraduates  of  the  University  of 
Southern  California.  They  received  credit  for  a  class  requirement  for 
participation. 
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TABLE  1 

DESCRIPTIONS  OF  ATTRIBUTES 


Risks  of  a  major  catastrophe:  The  risks  associated  with  a  major  accident 
at  the  facility  (  fatalities,  short  and  long  teim  health  impacts, 
loss  of  land,  etc.). 

Operational  impacts  on  human  health:  The  short  and  long  tern  health  effects 
(cancer,  asthma,  genetic  disorders,  bronchitis,  etc.)  of  pollution  pro¬ 
duced  by  daily  operations  at  the  facility. 

Impacts  on  flora  and  fauna:  The  impact  of  the  new  facility  on  the  ecosystems 
of  various  life  forms ,  especially  those  of  endangered  species . 

Impacts  of  operations  on  environmental  resources:  The  impact  of  the  sug- 
gested  facility  on  the  surrounding  environment  (water,  noise,  air 
quality,  soil,  etc.)  due  to  normal  operations. 

Economic  growth:  The  impact  of  the  proposed  operation  on  potential  economic 
growth  in  providing  stability  of  both  the  private  sector  and  public 
institutions  (e.g.,  business  responsiveness,  increased  availability 
of  consumer  goods  and  services) . 

Social  and  political  impacts :  Changes  in  local,  national,  and  international 
power  balance  (including  consequences  of  proliferation  and  of  increased 
power  of  experts,  business,  and  political  groups). 

Impacts  on  hunan  resources:  The  impact  of  the  proposed  operation  on 

accessibility  and  preservation  of  recreational,  cultural,  historical, 
and  archeaological  reserves  surrounding  the  facility. 

Psychological  impacts:  The  psychological  impact  of  the  proposed  facility 
on  the  surrounding  communities  (fears,  worries,  concerns  of  parents, 
etc.).  Alienation  of  the  community  from  leadership  (government, 
experts ,  business ,  etc . ) . 

Day  to  day  system  reliability:  The  adequacy  of  the  proposed  system  to 

supply  basic  energy  requirements  and  preventing  brownouts,  blackouts, 
and  loss  of  load. 

Overall  system  stability  and  efficiency:  The  ability  of  the  proposed 

operation  to  adapt  to  diversity  of  fuel  supplies  and  co-generation 
requirements.  Also  included  are  considerations  of  adequacy  of  the 
scale  to  meet  the  projected  needs  of  the  community,  and  the  time 
scale  to  commission  and  subsequent  decommission  of  the  operation. 

Direct  electricity  costs --consumption:  The  impact  of  the  proposed 
facility  on  operational  and  short  term  costs  that  are  to  be 
absorbed  by  the  consuner  through  the  projected  increase  of  the 
t/KWh  charge. 
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Sales  and  returns:  The  impact  of  the  prcoosed  facility  on  sales  and 
returns  to  the  utility  company. 

Indirect  electricity  costs:  The  impact  of  the  proposed  operation  on 
costs  associated  with  shutdowns,  waste  storage,  reclamation, 
regulation,  accidents,  brownouts,  blackouts,  energy  conversions, 
and  on-going  energy  research  and  development. 
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Procedure .  Subjects  were  run  individually.  They  were  familiarized 
with  the  energy  scenario,  the  value  tree,  and  the  options  in  a  booklet 
describing  the  problem.  Attribute  definitions  were  provided  on  cue  cards, 
so  that  subjects  could  refer  to  them  throughout  the  task.  Subjects  were 
instructed  that  responses  should  reflect  their  own  beliefs  and  preferences 
about  risks  and  benefits  of  alternative  sources  of  electricity  supply 
and  thus  there  were  no  right  or  wrong  answers.  They  required  about  lh  hours 
to  complete  all  tasks. 

The  subjects'  first  task  was  to  rate  each  of  the  three  energy  supply 
options  at  each  of  the  22  nodes  of  the  value  tree,  starting  with  13  twig 
level  attributes  and  moving  up.  Finally,  each  subject  made  an  overall 
rating  of  the  value  of  the  three  supply  options.  All  ratings  were  made 
in  the  following  way:  Subjects  first  decided  which  option  was  the  best 
on  an  attribute  and  gave  it  a  value  of  100.  Next,  they  decided  which  option 
was  worst  on  that  attribute,  giving  it  a  value  of  0.  The  remaining  option 
was  placed  somewhere  between  0  and  100,  such  that  its  position  expressed 
their  belief  of  the  standing  of  that  option  relative  to  the  other  two. 

Subjects  next  performed  either  the  hierarchical  or  non-hierarchical 
weighting  task.  The  order  of  these  tasks  was  randomized  across  subjects. 

The  hierarchical  procedure  asked  subjects  to  judge  importance  ratios  for 
pairs  or  triplets  of  attributes  as  dictated  by  the  structure  of  the  value 
tree.  For  instance,  each  subject  was  asked  to  compare  "Risk  of  a  major 
catastrophe"  with  "Operational  impacts  on  health".  Attributes  were  ranked,  the 
least  important  attribute  given  a  value  of  10,  and  the  other  attributes 
given  values  preserving  the  subjects'  judgment  of  the  ratios  of  importance 
weight.  Judgments  were  required  at  ten  such  nodes  in  the  tree  (8  pairs 
and  2  triplets) . 
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The  same  ratio  judgment  procedure  was  followed  for  non- hierarchical 
weighting,  except  that  subjects  were  given  all  thirteen  lowest  level 
attributes  at  the  same  time,  and  they  rank-ordered  and  weighted  the 
entire  list  simultaneously. 

The  experimenter  then  calculated  weights,  normalized  to  sun  to  1.0, 
for  each  of  the  13  lowest  level  attributes  from  the  judgments  fran  both 
the  hierarchical  and  non-hierarchical  assessment  procedures.  These  were 
presented  as  a  pair  of  lists  side  by  side  with  no  identification  of  which 
list  came  from  which  procedure.  The  subjects  were  then  asked  to  determine 
a  final  set  of  normalized  weights.  They  were  told  that  they  could  choose 
either  of  the  two  lists  or  construct  another  final  weight  list,  the  only 
restriction  being  that  the  final  list  sum  to  1.0. 

TV.  Results 

The  first  part  of  our  analysis  compared  the  variances  of  the  hier¬ 
archical  and  non-hierarchical  weights.  For  33  (89%)  of  the  37  subjects 
hierarchical  weights  had  a  higher  variance  than  non-hierarchical  weights. 
For  30  subjects  (81%)  the  variance  of  hierarchical  weights  was  more  than 
double  the  variance  of  non-hierarchical  weights. 

The  procedure  for  eliciting  final  weights  was  unlike  the  other  two; 
it  simply  asked  subjects  to  give  a  "proportion"  of  the  total  weight 
to  each  dimension.  Thus,  judgments  were  of  the  "constant  sum"  rather 
than  ratio-estimation  type.  In  order  to  compare  the  ordinal  properties 
of  the  three  weight  sets,  we  computed  Kendall  tau  rank  order  correlations 
for  each  pair  of  weight  vectors  and  for  each  subject.  The  hierarchical 
and  non-hierarchical  weight  vectors  showed  only  modest  ordinal  agreement 
(average  tau:  .57).  Final  weights  corresponded  more  closely  to  the 
hierarchical  weights  (average  tau:  .72)  than  to  non-hierarchical  weights 
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(average  tau:  .62).  Since  tau  is  the  proportion  o£  identical  pairwise 
rankings  minus  the  proportion  of  inverted  pairwise  rankings,  we  can 
conclude  that  there  were  approximately  10%  more  inversions  between  non- 
hierarchical  and  final  weights  than  between  hierarchical  and  final  weights. 

Next  we  examined  the  ability  of  five  different  models  to  predict 
holistic  rank  orders  of  alternatives.  The  first  three  models  combined 
the  three  weighting  schemes  (hierarchical,  non-hierarchical,  final) 
with  first  level  ratings;  the  fourth  model  used  second  level  ratings  and 
(hierarchical)  second  level  weights;  and  the  fifth  model  combined  third 
level  ratings  with  (non-hierarchical)  third  level  weights.  The  main 
results,  summarized  in  Table  2,  is  that  there  exists  only  moderate  con¬ 
sistency  between  holistic  rankings  of  alternatives  and  rankings  implied  by 
the  composites  of  the  five  multiattribute  utility  models.  Different 
weighting  techniques  produced  about  the  same  amount  of  inconsistency, 
and  there  is  no  improvement  for  higher  level  models. 

Insert  Table  2  about  here 

We  next  examined  attribute  weights  and  ratings  conditional  on 
subjects*  holistic  first  choice.  We  identified  three  groups:  the 
"PRO-NUC(lear)*'  group  (n=10)  gave  nuclear  the  higher  overall  rating, 
the  "PRO-COAL"  group  (n=12)  rated  coal  the  highest,  and  the  "PRO-CONS (ervation)" 
group  (n=15)  named  the  geothermal/conservation  package  as  their  preferred 
option.  Table  3  lists  the  average  weights  for  the  three  groups  and  for 
the  hierarchical  and  non-hierarchical  weighting  procedures.  These  aver¬ 
age  weights  show  substantial  agreement  across  groups.  For  example,  all 
groups  considered  Heal th/Safety /Environment  as  the  most  important  factor 
by  far.  Finances /Reliability/Stability  and  Political/Social/Psychological 
were  a  distant  second  and  third.  None  of  the  weight  differences  among 


OF  ALTERNATIVES  AND  MODEL  PREDICTIONS 


j 

k. 


groups  were  significant. 
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Insert  Table  3  about  here 

Since  subjects  weight  the  attributes  about  the  same,  no  matter  wh  .t 
option  is  holistically  preferred,  it  would  seem  that  the  single  attribute 
ratings  of  the  options  must  differ  to  explain  group  differences  in  holistic 
ratings.  Table  4  shows  group  averages  of  the  ratings  of  the  three  alter¬ 
natives  on  all  branches  and  nodes  of  the  tree.  Consider  the  Health/Safety/ 
Environment  branch  first  (Table  4a).  In  this  table  we  underlined  for  each 

Insert  Tables  4a-c  about  here 

group  the  mean  rating  that  was  highest  across  options  (columns).  As  is 
evident,  there  was  considerable  agreement  among  the  three  groups  that  the 
Conservation/Geotheimal  option  was  best  with  respect  to  all  Health/Safety/ 
Environment  attributes.  We  also  marked  with  an  asterisk  the  mean  rating 
that  was  highest  across  groups  (rows).  This  analysis  reveals  a  trend  for 
a  group  to  favor  the  option  that  it  prefers  holistically.  To  substantiate 
these  trends,  a  two-way  ANOVA  was  performed  on  each  3x3  matrix  in  Table  4a. 
As  one  would  expect,  the  group  main  effect  was  insignificant,  while  the 
option  main  effect  was  significant  for  all  six  matrices.  The  group  bias 
to  favor  its  own  option  showed  as  a  significant  group  by  option  interaction 
in  the  "Catastrophe"  attribute  only. 

A  similar  pattern  emerged  in  the  Finances/Stability /Reliability 
branch  (Table  4c) .  Here  groups  agreed  that  nuclear  or  coal  were  the 
best  options,  while  conservation  scored  lowest.  These  trends  appeared 
as  significant  option  main  effects  in  all  but  the  direct  and  indirect 
cost  attributes.  The  bias  to  favor  the  holistically  preferred  option 
also  emerged  as  a  significant  option  by  group  interaction  for  the 
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RATINGS  OF  ALTERNATIVES  BY  GROUP  FOR  THE  FINANCES/STABILITY/RELIABILITY  BRANCH 
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Stability  /Reliability  attributes.  In  the  Political/Social/Psychological 
branch  the  three  groups  shew  some  agreement  that  conservation  is  best 
in  all  but  the  "economic  growth"  attribute  where  all  groups  rated 
nuclear  the  highest.  All  attributes  except  "political  impacts"  show  a 
significant  option  main  effect.  The  option  by  group  interaction  is  significant 
cnly  for  the  "psychological  impacts"  attribute  and  for  the  general  value 
"Political/Social/Psychological" . 

In  summary,  we  found  that  the  different  groups,  defined  by  their  holistic 
first  choices,  tended  to  show  remarkable  agreement  on  their  attribute-wise 
first  choices.  Nevertheless,  we  also  detected  a  tendency  in  several 
attributes  for  the  groups  to  favor  the  option  that  they  prefer  holistically. 

To  further  clarify  the  differences  and  similarities  among  the  three 
groups,  we  calculated  each  subject’s  utilities  for  the  three  options  based 
on  the  five  possible  multiattribute  utility  models.  The  first  row  of 
Table  5  shows  the  average  holistic  rating  for  each  group;  rows  2-6  show  the 
average  multiattribute  utilities.  Note  that  we  cannot  directly  compare 

Insert  Table  5  about  here 

the  scale  of  the  calculated  utilities  with  the  holistic  scale,  since  the 
latter  was  forced  to  go  through  the  extreme  points  0  and  100.  However, 
three  of  the  five  multiattribute  utility  models  of  the  PRO-NUC  group  favor 
the  Conservation  option,  and  are  therefore  inconsistent  with  that  group's 
holistic  choice.  Further,  the  PRO-COAL  group’s  multiattribute  utilities 
for  coal  and  conservation  are  very  close.  Only  the 
multiattribute  utilities  of  the  PRO-CONS  group  clearly  follcw  the  holistic 
pattern.  Composites  from  higher  level  models  do  not  resemble  holistic 
ratings  more  closely  than  those  of  lower  level  models. 
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To  further  investigate  this  apparent  inconsistency  between  holistic 
judgments  and  multiattribute  models,  we  compared  model  predictions  of  each 
subject's  first  choice  and  full  rank  order  with  holistic  assessments 
separately  for  each  group.  The  striking  result,  summarized  in  Table  6, 
is  that  there  exists  substantial  inconsistency  for  individuals  preferring 
the  nuclear  option,  less  for  those  preferring  coal,  and  very  little  for 
those  preferring  conservation. 

Insert  Table  6  about  here 


V.  Discussion 

This  experiment  had  three  basic  findings.  First,  it  replicated 
Sayeki  and  Vesper's  (1973)  result  that  hierarchical  weights  are  steeper 
than  nan-hierarchical  weights.  Second,  all  multiattribute  models  shewed 
considerable  discrepancies  from  holistic  rankings,  independent  of  the 
weighting  procedure  or  the  level  of  the  tree  at  which  alternatives  were 
rated.  However,  the  degree  of  inconsistency  depended  on  the  subjects' 
first  holistic  choice.  Third,  groups  identified  by  their  holistic  first 
choice  had  very  similar  multiattribute  models.  There  was  little  disa¬ 
greement  about  weights  and  only  moderate  disagreement  about  the  attribute 
ratings  of  the  energy  options,  in  spite  of  the  obvious  disagreement  in 
holistic  preferences. 

One  interpretation  of  the  flatter  non-hierarchical  weights  is  that 
subjects  have  a  response  bias  against  large  weight  ratios.  Since  rela¬ 
tively  flat  higher  level  weights  can  still  produce  steep  lower  level 
weights,  this  response  bias  would  not  have  affected  hierarchical  weights 
as  much  as  non-hierarchical  weights. 


TABLE  6 

CONSISTENCY  BETWEEN  MULTIATTRIBUTE  ORDERINGS 
AND  HOLISTIC  PREFERENCES  BY  GROUP 


'Wnwk, 


-23- 

There  are  several  reasons  why  subjects'  holistic  rankings  and 
their  multi attribute  utility  models  were  only  moderatley  consistent. 

First,  the  value  tree  in  Figure  2  may  not  have  represented  individual 
subjects'  values;  second,  ratings  may  have  been  distorted  or  otherwise 
in  error;  and  third,  elicited  weights  may  have  differed  from  the  implicit 
weights  underlying  holistic  rankings. 

The  value  tree  in  Figure  2  was  constructed  by  pruning  and  combining 
separate  trees  that  reflected  the  values  of  several  political  actors  in 
the  energy  debate.  The  combined  tree  therefore  provided  a  rather  broad, 
balanced,  and  social  view  of  the  problem.  Had  we  elicited  value  trees 
from  each  subject,  we  probably  would  have  found  many  idiosyncratic  and 
politically  motivated  structures.  While  holistic  rankings  allowed  subjects 
to  engage  such  idiosyncratic  perspectives,  the  value  tree  may  have  created 
a  common  viewpoint  that  masked  individual  factors,  while  making  the  social 
factors  more  relevant. 

Ratings  were  another  potential  source  of  inconsistency.  Our  subjects 
were  not  experts  in  assessing  energy  options,  and  they  had  to  rely  on 
whatever  knowledge  they  possessed  when  rating  options  on  the  attributes. 
While  subjects  had  sane  knowledge  (or  at  least  opinions)  about  the 
relative  standing  of  the  three  alternatives  on  most  attributes,  they 
were  probably  not  very  sure  of  their  judgments  in  some  attributes, 
and  produced  very  tentative  ratings.  One  would  expect  that  subjects 
discount  such  "weak"  ratings  when  making  holistic  comparisons  among 
alternatives.  In  principle,  the  multiattribute  models  could  achieve 
the  same  effect  by  reducing  the  weight  for  an  attribute  about  which  a 
subject  has  little  knowledge.  But  if  subjects  interpreted  weights  in  ternis 


of  the  general  social  relevance  of  an  attribute,  such  discounting  would 
not  necessarily  have  occurred.  Multiplying  relatively  whimsical  ratings 
with  undiscounted  "social  relevance”  type  weights  would  tend  to  produce 
inconsistencies  with  holistic  judgments. 

Neither  idiosyncratic  value  structures  nor  errors  in  ratings  can 
explain  the  finding  that  the  three  groups  (PRD-NUC,  PRO-COAL,  PRO- CONS) 
showed  substantial  differences  in  the  consistency  between  holistic 
rankings  and  the  multiattribute  models.  A  possible  explanation  is  that 
groups  perceived  the  value  ranges  in  the  attributes  differently.  For 
example,  all  groups  rated  the  geotheimal/conservation  option  highest 
on  the  Heal th/Safety /Environment  attribute ,  coal  being  second  in  most 
attributes,  nuclear  third.  But  the  PRO-NUC  group  may  have  perceived  the 
differences  among  options  in  the  Health/Safety /Environment  attributes 
as  very  small.  The  PRO-CONS  groups,  on  the  other  hand,  may  have  per¬ 
ceived  very  large  differences  in  this  set  of  attributes.  The  multi¬ 
attribute  utility  models  supposedly  compensate  for  such  differences  in 
perceived  value  ranges  by  an  appropriate  adjustment  of  attribute  weights. 
Proper  adjustment  of  the  weights  would  have  required  to  lower  weights 
for  small  perceived  value  ranges,  and  increase  than  for  large  ones.  If, 
instead,  importance  judgments  were  range  insensitive,  reflecting  a  general 
social  concern  rather  than  an  appropriate  re-scaling  of  attributes,  the 
multiattribute  utility  models  would  be  distorted. 

As  mentioned  above,  the  instructions  for  weight  elicitation  could  have 
been  interpreted  by  subjects  as  instructions  to  assess  the  general  social 
relevance  of  the  attributes.  This  interpretation  is  supported  by  the 
finding  that  most  subjects,  regardless  of  group,  assigned  about  50%  of 
the  weights  to  the  socially  relevant  Health/Safety /Environment  attributes. 


Assuming  that  the  PRD-NUC  subjects  perceived  the  value  range  in  these 
attributes  as  relatively  small,  their  weights  should  have  been  adjusted 
downward.  Failure  to  adjust  would  artificially  increase  the  multi¬ 
attribute  utilities  of  the  conservation/geothermal  option  which  scored 
highly  on  these  attributes.  Such  an  increase  would  be  in  conflict  with 
the  holistic  preferences  of  the  PRO-NUC  group  and  explain  the  high 
rate  of  inconsistency  for  that  group. 

The  convergence  of  multiattribute  models  across  groups  was  previously 
reported  by  Aschenbrenner  and  Kasubek  (1978)  and  by  Gardiner  and  Edwards 
(1976).  This  could  be  an  extremely  relevant  finding,  if  it  was  indeed 
due  to  the  convergence  of  view  when  formulated  in  a  more  rational  and 
cognitive  manner.  It  points  to  the  usefulness  of  multiattribute  models 
for  aiding  discussions  and  negotiations  between  groups  involved  in  con¬ 
troversial  and  political  decisions.  These  models  could  be  built  separately 
for  each  adversary  group  to  pinpoint  the  sources  of  agreement  and  disa¬ 
greement  and  the  need  for  additional  information  or  new  options.  From 
this  perspective  our  results  indicate  that  the  resolution  of  inconsistencies 
within  groups  may  be  as  important  as  the  resolution  of  disagreement  across 
groups.  As  in  our  study,  such  disagreements  may  be  much  smaller  than  the 
initial  controversy  might  indicate.  In  particular,  we  found  virtually 
no  disagreements  about  weights,  and  only  some  about  ratings  of  alternatives. 

Unfortunately,  we  cannot  exclude  the  possibility  that  the  convergence 
of  the  multiattribute  models  was  due  to  an  artifact  of  range  insensitive 
importance  weights,  or  perhaps  due  to  distortions  of  weights  and  ratings 
towards  "socially  desirable"  values.  The  obvious  experiment  would  con¬ 
front  subjects  with  the  inconsistencies  between  their  holistic  ratings 
and  their  multi  attribute  models  and  ask  them  to  resolve  them. 
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